Structural traffic analysis for network security monitoring

Traffic on the Internet is constantly growing more complex and multifaceted. This natural evolution is mirrored by novel kinds of malicious traffic: automated attacks subvert thousands of machines at a time, enabling a wide range of subsequent attacks and nuisances such as distributed denial-of-service attacks and generation of vast amounts of unsolicited electronic mail. Consequently, there is a strong need to be able to tell malicious traffic from the benign. In this dissertation, I take several steps toward this goal. By leveraging structural aspects of network traffic, typical as well as malicious activity on computer networks can be fingerprinted and contrasted. A first avenue is the analysis of application-level flow content. I investigate the suitability of biological sequence alignment algorithms in the adversarial environment of the networking domain, and introduce a novel algorithm that is well over an order of magnitude faster than the commonly used Smith-Waterman algorithm while maintaining much of its flexibility. I introduce a novel and highly flexible model of traffic content based on sequence alignment, Common Substring Graphs, and demonstrate its versatility in a study of application-level protocol classification. Switching focus to the malicious, I pioneer the use of honeypots and sequence analysis algorithms for automated fingerprinting of malware and thus demonstrate the feasibility of fully automated network-based malware signature generation. I propose a second approach to fingerprinting the malicious: Packet Symmetry focuses on network-level structure and leverages the intuition that well-behaved applications do not transmit vastly more packets than they receive. Traffic analysis confirms the feasibility of employing packet symmetry for edge-based, ingress-focused prevention strategies for volume-based attacks.